Skip to content

Stop RedisHandler blocking logging calls and holding back buffered records - #742

Merged
RazerM merged 2 commits into
mainfrom
fix/redis-handler-lifecycle
Nov 9, 2026
Merged

RazerM merged 2 commits into
mainfrom
fix/redis-handler-lifecycle

Conversation

@RazerM

@RazerM RazerM commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

RedisHandler pushed to Redis while holding its lock, so logging calls waited whenever a push to an unreachable Redis was in progress. A failed push killed the flush thread. After that, records went out only when the buffer held exactly flush_threshold of them, and once a push there had failed, not until close(). disable_buffering() also left buffered records until close(), and closing during a flush could push records twice.

While buffering, the flush thread is now the only pusher. It holds no lock during the push and keeps a failed batch for the next attempt. The next logging call reports the first failure in a row through the handler's handle_error(). close() and disable_buffering() push what is buffered and raise if that fails. In a forked child, the handler starts a new flush thread when it first logs.

disable_buffering() left the buffered records in place, and emit()
pushed only when the buffer held exactly flush_threshold records. Once
the threshold dropped to 1, records stayed buffered until close().
RedisHandler pushed while holding its lock, so logging calls waited
whenever a push to an unreachable Redis was in progress. A failed push
also killed the flush thread, and closing during a flush could push
records twice.
@codspeed

codspeed Bot commented Nov 9, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ 3 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 38 untouched benchmarks


Comparing fix/redis-handler-lifecycle (c060ab2) with main (a661604)

Open in CodSpeed

@codecov

codecov Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.43182% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.65%. Comparing base (a661604) to head (c060ab2).

Files with missing lines Patch % Lines
src/logbook/queues.py 98.33% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #742      +/-   ##
==========================================
+ Coverage   91.40%   91.65%   +0.25%     
==========================================
  Files          49       49              
  Lines        7931     8098     +167     
  Branches      609      619      +10     
==========================================
+ Hits         7249     7422     +173     
+ Misses        546      542       -4     
+ Partials      136      134       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@RazerM
RazerM merged commit c1705d4 into main Nov 9, 2026
51 of 52 checks passed
@RazerM
RazerM deleted the fix/redis-handler-lifecycle branch November 9, 2026 23:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant